Adaptive Consistency Management for In-memory Storage

The availability of storagemedia with high capacity at low prices has recently increased the demand for software applications that are able to analyze large data volumes. Engineers build large-scale storage systems using both scale-up and scale-out techniques. Scale-up increases the amount of data a single nodes stores, whereas scale-out aggregates the capacity of several servers and increases the peak throughput of data transfers. Scale-out systems do not share any resources except for a communication bus, such that the participating compute nodes need to share information explicitly. High-speed communication in local-area networks and in-memory storage of information reduce the accesses latency compared to storage on harddisks. Traditional applicationdesigns are oftenunable to use large-scale storage efficiently. Parallelization of sequential programs faces problems of data interdependencies, distribution unawareness and errorproneness in distributed settings. Design patterns that are successful for sequential applications often do not apply anymore for concurrent execution. The success of a storage service depends largely on the acceptance by application developers. Thus, such a service must adhere to convenient design thinking while at the same time it should not degrade the performance of optimized applications. This thesis presents novel approaches to accommodate application programming through appropriate design of the storage service. By combining techniques from storage replication, peer-to-peer computing and optimistic synchronization, the proposed storage service relieves application programmers from handling failures and explicit lock management. Optimizations that are mostly transparent for the application allow to reduce false sharing effects and to increase storage utilization. Specifically, the thesis details the design and implementation of two adaptive techniques to improve the performance of distributed transactional memory. Adaptive replication makes storage objects available rapidly and increases update throughput by analyzing object access patterns. Adaptive conflict granularity allows bulk object transferswhile at the same timedetecting and avoiding false sharing situations. The described techniques simplify application programming by improving the context-awareness of distributed storage services. This thesis also introduces a framework for in-memory applications that adhere to the MapReduce programming model. The use and applicability of the suggested enhancements for a scalable storage service are exemplified with a number of applications from diverse problem domains including computer graphics, statistics and data mining. The examples also serve to analyze the performance and scalability of the storage service. The measurements demonstrate that the extensions improve the access parallelism of in-memory storage without complicating the programmingmodel or increasing storage requirements. In summary, this thesis presents several contributions to the research field of large-scale in-memory data management. The evaluation of the contributions proves their applicability and potential for realistic workload.